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CHAPTER  I 


A  STATEMENT  OF  THE  PROBLEM: 

THE  PURPOSE  OF  THE  INVESTIGATION 

At  the  conclusion  of  the  marking  of  the  candidates 1 
papers  in  the  English  3  Examination  which  was  conducted  by 
the  Department  of  Education  of  the  Province  of  Alberta  in 
June  1943,  the  committee  in  charge  was  somewhat  disappointed 
over  the  results  obtained.  Despite  the  refinements  in  exam¬ 
ination  techniques  which  had  been  adopted  to  ensure  an 
adequate  discrimination  between  the  abilities  of  the  various 
candidates,  the  committee  found  on  examining  the  distribution 
of  the  results  that  there  was  an  inadequate  spread,  that  is, 
that  the  results  tended  to  be  "bunched”  together.  In  fact, 
it  was  the  opinion  of  the  committee  that  such  a  degree  of 
"bunching”  might  tend  to  invalidate  the  examination  as  a 
discriminating  measure  of  ability.  With  a  very  narrow  range 
of  scores  the  possibility  of  dividing  the  candidates  into 
the  categories  H,  A,  B,  C,  and  D  with  any  degree  of  relia¬ 
bility  is  rather  small,  since  as  the  range  becomes  smaller, 
the  frequencies  of  each  class  interval  become  correspondingly 
larger. 

With  few  class  intervals  and  large  class  frequencies, 
it  is  quite  apparent  that  it  is  difficult  to  subdivide  the 
population  into  categories  that  represent  significant  dif- 
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ferences.  This  seemed  to  be  the  point  to  which  the  English 
examination  results  were  approaching. 

The  committee  rather  felt  that,  in  view  of  the 
objectivity  of  the  examination  and  of  the  other  devices  used, 
the  distribution  of  results  should  be  more  nearly  like  that 
obtained  in  the  mathematical  subjects. 

For  the  sake  of  comparison,  in  addition  to  the  graphs 
of  distributions  for  English  during  the  past  three  years, 
there  are  here  presented  the  graphs1  of  the  distributions 
for  Physics  2,  Algebra  2  and  Social  Studies  3  in  the  year 
1943. 


!The  Graphs  shown  here  are  made  from  the  actual  charts 
drawn  for  Department  of  Education,  Alberta. 
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Fig.  1.  -  Frequency  Distribution  of  English  3 
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Figures  1,  2,  and  3  represent  graphically  the  fre¬ 
quency  distributions  of  the  results  of  the  English  3 
examinations  for  the  years  1943,  1942  and  1941  respectively. 
From  these,  the  following  information  in  Table  I  is  easily 
observable . 


TABLE  I 

RANGES  OF  SCORES  ON  ENGLISH  3  FOR  THREE  YEARS 


1941 

1942 

1943 

Possible  range  of  scores 

0-100 

0-100 

0-100 

Actual  range  of  scores 

21-85 

19-85 

27-87 

Percent  Actual  of  Possible 

65 

67 

61 

It  will  be  noted  that  the  actual  range  of  scores  is 
less  than  two- thirds  of  the  possible  range.  However,  from 
the  graphs  it  is  quite  noticeable  that  the  scores  of  a  very 
large  number  of  candidates  lie  within  a  much  smaller  range. 

On  the  other  hand,  Figures  4,  5  and  6,  which  show  the 
frequency  distributions  for  Social  Studies  3,  Physics  2,  and 
Algebra  3,  respectively  for  the  year  1943,  display  much  wider 
ranges  with  no  abnormal  bunching  at  any  part  of  the  range. 

The  following  data  are  noticeable  in  contrast  with 
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Fig.  4  -  Frequency  Distribution  of  Social  Studies  5, 


1943, 
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Fig.  5.  -  Frequency  Distribution  of  Physics  2 
Fig.  6.  -  Frequency  Distribution  of  Algebra  5 
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1943. 
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the  data  presented  in  Table  I. 

TABLE  II 

RANGES  OF  SCORES  IN  OTHER  SUBJECTS 


Soci  al 
Studies  3 
1943 

Physics  2 

1943 

Algebra  2 
1943 

Possible  range  of  scores 

0-109 

0-160 

0-165 

Actual  range  of  scores 

21-93 

6-153 

10-160 

Percent  Actual  of  Possible 

67$ 

97$ 

91.5$ 

In  an  effort  to  determine  the  cause  of  this  alleged 
distortion,  the  examination  results  were  inspected.  It  was  no 
ticed  that  the  marks  assigned  by  the  instructors  for  the 
candidates1  tern  reading,  that  is,  the  Part  I  A  Score}  were 
in  general  fifteen  to  thirty  percent  higher  than  the  marks 
obtained  by  the  same  candidates  on  the  Department’s  written 
test  in  Literature,  that  is,  on  the  Part  1  B  Score.  Of  neces¬ 
sity,  the  teachers’  marks  could  not  parallel  the  marks  earned 
by  the  students  on  the  written  test,  for  as  the  marks  on  the 
Part  I  B  Test  reached  the  higher  percentages  the  teachers* 


iThe  regulations  of  the  Department  of  Education  describing 
the  format  of  the  English  3  examination  are  given  in  Appendix 
A,  page  81. 
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marks  attained  the  upper  limit  of  100%  beyond  which  they  could 
not  go.  Hence,  the  effect  of  the  teachers’  marks  would  seem 
to  be  to  raise  the  lower  percentages  and  at  the  same  time  keep 
the  upper  percentages  stationary.  This  is  precisely  what  the 
graphs  seem  to  indicate,  namely,  that  the  lower  percentages 
were  being  displaced  from  their  normal  position  and  thrown  in 
with  those  higher  in  the  scale. 

The  problem  then  to  be  investigated  is  as  follows: 

"IS  THE  TEACHERS’  PART  I  A  SCORE’  THE  CAUSE  OF  THE  INADEQUATE 
SPREAD  AND  OF  THE  "BUNCHING”  OF  THE  RESULTS?"  However,  other 
effects  as  they  preseht  themselves  mil  be  studied. 


i 


a!  ;  :':  r;  ■■  :nla j:h~ 

1  :  •  •  :  ' 


. 

. 

.'J  '  :  :-S 


.  ■  ■  J—  :■  J  ?.  '  ;  ;  .  :j!  'y.-o 

■ 


- 


: 


CHAPTER  II 


METHOD  OF  THE  INVESTIGATION 


Through  the  kindness  of  Mr.  B.E.  Walker  and  his  staff 
of  the  Examinations  Branch,  Department  of  Education,  the  re¬ 
cords  of  all  the  students  who  wrote  the  Departmental  Exam¬ 
inations  in  lune  1943  were  made  available.  These  records, 
as  far  as  the  English  3  and  Social  Studies  3  are  concerned, 
were  transcribed  onto  cards  specially  printed  for  the  purpose. 
A  sample  card  filled  in  with  the  actual  marks  of  a  student 
is  shown  below. 


Dept’l 
_ Number _ 

Student 

Municipality 

Our  File 
Number 

English  3 

Lit. 

A 

75  ' 

S' 

53 

7 

108 

s 

54 

Tr. 

Score 

Grade 

Score 

—A 

Tr. 

Score 

Conf. 

Note 

Lang 
Qu.  5 

Lit. 
Qu.  5 

B 

30  * 

1 

53 

B 

85  " 
43'" 

'3 

44 

C* 

is 

75 

iC 

vi 

V 

£ 

Lang. 

11  5; 

55  ^ 

44  * 

Social  Studies  3 

it- 

36 

n 

10 

ZV 

46 

z/ 

41 

a.  z. 

c 

i  j 

Fig. 7.  -  A  sample  of  card  used  in  recording  the  data  for 
each  candidate. 
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An  explanation  of  the  different  marks  listed  on  the 
card  follows: 

(1)  This  is  the  mark  assigned  by  the  teacher  based 
on  the  students  reading  and  study  of  the  prescribed  texts  - 
maximum  100  -  and  is  deferred  to  as  the  Part  I  A  Score  or 
the  Literature  A  mark. 

(2)  This  is  the  mark  obtained  by  the  candidate  on 
the  Departments  written  test  on  Literature  -  maximum  100  - 
and  is  referred  to  as  the  Literature  B  mark. 

(3)  This  is  the  Language  A  mark  and  is  the  mark  as¬ 
signed  to  Question  5  of  the  Literature  paper  for  its  com¬ 
position  value  by  a  separate  committee  using  a  special 
language  marking  technique  -  maximum  30. 

(4)  The  mark  for  the  written  language  test  -  maximum  70. 

(5)  The  combined  literature  mark.  Marks  1  and  2  are 
added  together  and  divided  by  2,  giving  a  maximum  of  100%. 

(6)  The  combined  mark  for  language.  Marks  3  and  4 
are  added  together,  giving  a  possible  maximum  of  100%. 

(7)  The  total  mark  for  English.  This  is  obtained 
by  adding  marks  5  and  6  together  -  possible  maximum  200. 

(8)  This  is  the  nr aw- score"  and  is  obtained  by 
halving  the  marks  in  .7,  Maximum  100.  It  is  from  the  marks 
in  this  space  that  the  raw  score  distribution  Is  made. 

(9)  The  Transmuted  Score.  This  is  obtained  from  8 
by  using  a  transmuting  scale  which  changes  the  range  of  the 
raw  scores  to  a  range  0-100.  Thus  the  highest  mark  In  8 
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becomes  100$  and  the  lowest  becomes  0.  This  is  the  mark 
of  -which  the  candidate  receives  notification, 

(10)  The  grade-letters,  H,A,B,C,D,  according  to  the 
marks  in  9.  H  =  80  plus,  A  s  65  plus,  B  s  50  plus,  C  =  40 
plus,  D  »  less  than  40, 

(11)  This  is  the  investigator’s  mark,  called  ’’Score 
-  A”,  and  is  the  total  of  marks  6  and  2.  The  Literature  A 
mark  has  been  left  out  of  the  computation.  Maximum  200. 

(12)  Mark  11  is  halved  giving  a  maximum  100. 

(13)  Using  a  similar  scale  to  that  used  by  the  De¬ 
partment  in  9,  the  marks  in  12  are  transmuted  to  a  range 

0  -  100. 

(14)  Grade  letters  -  (as  in  10  above)  applied  to 
the  marks  in  13. 

(15)  Confidential  mark.  This  is  a  reference  mark 
only,  and  is  assigned  by  the  Principal  of  the  school  to 
indicate  his  opinion  of  the  candidate’s  work  during  the  year. 
It  may  also  be  viewed  as  the  Principal’s  prediction  of  the 
candidate’s  score  for  the  examination.  It  does  not  enter 
into  the  computation. 

(IS)  The  mark  for  Question  5  of  the  Language  paper, 
that  is,  for  the  construction  of  a  paragraph  of  prose.  The 
evaluating  committee  is  the  same  as  marked  for  3,  and  the  tech- 
nique  used  is  the  same.  Maximum  10. 

(17)  The  mark  for  Question  5  of  the  Literature  paper. 
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This  is  a  question  to  test  the  candidate’s  knowledge  of  the 
canons  of  literature  and  his  appreciation.  The  question  is 
marked  now  for  literary  content  in  distinction  to  the  mark 
given  in  space  3  which  was  the  language  value  of  the  same 
writing.  Maximum  for  the  literature  answer  is  10. 

(18)  Mark  for  Social  Studies  content  of  the  Social 
Studies  paper.  Maximum  80. 

(19)  Mark  for  the  Language  value  of  the  Social 
Studies  answers.  Maximum  20. 

(20)  Total  Raw-Score  for  Social  Studies  -  Marks  18 
and  19  are  added. 

(21)  Transmutation  of  20  to  a  range  -  0  -  100. 

(22)  Grade  letters  of  21,  similar  to  10  above. 
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Dept’l 

Number 


Student 


Municipality 


Our  File 
Number 


English  3 

Lit. 

A 

B 

100  ' 
100  * 

r 

100 

7 

200 

t 

100 

Tr. 

Score 

Grade 

Score 

—A 

Tr. 

Score 

Conf. 

Note 

Lang 
Qu.  5 

Lit. 
Qu.  5 

10  0 

'0 

H, 

200 

100 

10C 

u 

H 

1 

100 

10 

'7 

10 

Lang. 

o 

o  * 

Social  Studies  3 

~  o 

CO 

n 

20 

a 

100 

2/ 

100 

h 

L  A 

Fig.  7b.  -  Card  used  for  rural  candidates  showing 
totals  for  each  part. 
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Two  colours  of  cards  were  used,  one  for  urban  students  the 
other  for  rural  students,  in  case  it  should  be  desirable 
to  study  the  effects  on  each  group  separately.  A  card  was 
made  for  each  of  the  3320  students  who  wrote  on  English  3. 

For  those  who  wrote  both  English  3  and  Social  Studies  3, 
the  marks  in  both  subjects  were  transcribed. 

A  sample  of  about  three  hundred  Literature  papers 
and  the  corresponding  language  papers  were  obtained.  From 
these  examination  papers  the  marks  of  Question  5  on  the 
Literature  ancl  Question  5  on  the  Language  were  obtained  and 
marked. on  the  cards. 

The  Confidential  Reports  of  the*  Principals  of  each 
school  were  obtained  from  the  Department.  From  these  reports, 
the  confidential  marks  of  the  candidates  who  wrote  the  sample 
examination  papers  which  were  being  used  were  obtained. 

Thus  the  complete  information  on  each  member  of  the 
distribution  population  was  available  to  the  investigator. 

Each  card  was  carefully  scrutinized  to  see  trends,  influ¬ 
encing  factors  etc. 

It  was  decided  that,  for  the  work  dealing  with  the 
primary  object  of  the  investigation,  the  complete  population 
would  be  used  while  for  the  study  of ."other  incidental  effects" 
such  as  the  correlations  between  the  various  parts -yof  the 
examination,  the  data  derived  from  the  samp le  would  be  used. 

A  graph  of  the  frequency  distribution  of  the  sample, 
(Figure  17)  is  shown.  It  will  be  seen  that  it  resembles  the 
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the  graph  of  the  distribution  of  the  whole  population.  The 
almost  complete  coincidence  of  the  two  distributions  is  more 
clearly  seen  from  the  ogives  in  Figure  16,  showing  the  cu¬ 
mulative  percentage  curve  of  the  two  distributions  as  well 
as  the  mean  and  medial  points.  Comparing  the  frequency 
distribution  of  the  sample  and  the  whole  population  in  res¬ 
pect  to  ’’raw  score”,  we  find: 


TABLE  III 

COMPARISON  OF  SAMPLE  AND  POPULATION 


Mean 

Median 

Sample 

54.3 

54 

Whole  Population 

54.2 

54.5 

Thus  it  will  be  seen  that  for  practical  purposes  the  sample 
is  representative  of  the  whole  population. 

From  there  on,  the  investigation  proceeds  by  making 
comparisons  of  distributions  and  measuring  correlations. 
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CHAPTER  III 


THE  EFFECT  OF  .  THE  PART  I  A  SCORE 
ON  THE  DISTRIBUTION  OF  THE  FINAL  SCORE 

The  principal  object  of  this  investigation  is  to  de¬ 
termine  whether  or  not  the  marks  given  by  the  teacher  for  the 
students1  work  in  the  reading  and  study  of  the  recommended 
texts,  that  is,  the  Part  I  A  Score  or  the  Literature  A  mark, 
is  the  cause  of  the  inadequate  spread  of  the  "raw  scores"  or 
of  any  abnormal  bunching  of  them  around  the  middle  of  the 
range.  In  other  words,  what  Is  needed  is  the  answer  to  the 
question, "what  effect  does  the  Part  I  A  mark  have  on  the  re¬ 
maining  portion  of  a  candidate’s  marks  (the  Score-A)  when  the 
two  are  combined  together  to  form  his  "raw- score"?" 

This  effect  can  be  considered  in  two  ways.  It  can  be 
viewed,  either  from  total  effects,  that  is,  the  effect  of  the 
distribution  of  Part  I  A  marks  on  the  distribution  of  the 
Score-A  marks  in  producing  the  distribution  of  the  "Raw  Scores", 
or  It  can  be  viewed  in  the  light  of  the  changes  made  in  the 
individual  scores.  In  this  chapter,  the  former  will  be ■ con¬ 
sidered,  while  the  latter  effect  will  be  discussed  In  the  next 
chapter. 

The  procedure  in  this  chapter  will  be,  first,  to  study 
the  distribution  of  the  "Raw  Scores";  secondly,  to  study  the 
distributions  of  the  components  of  the  "Raw  Scores"  namely  the 
Part  I  A  mark  and  the  Score-A  marks;  and  finally,  to  discuss  the 
mutual  effect  of  the  "Part  I  A  marks"  and  the'  "Score-A  marks  in 
producing  the  "Raw  Scores". 
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TABLE  IV 

ENGLISH  3:  DISTRIBUTION  OF  1943  RAW  SCORES 


Score 

MLd-Pt. 

Frequency 

j  Cumulative 

84-87 

86 

< 

2 

3316 

80-83 

82 

5 

3314 

76-79 

78 

26 

3309 

72-75 

74 

83 

3283 

63-71 

70 

132 

3200 

64-67 

66 

258 

3068 

60-63 

62 

436 

2810 

56-59 

58 

489 

2374 

52-55 

54 

586 

1885 

48-51 

50 

521 

1299 

44-47 

46 

385 

778 

40-43 

42 

236 

393 

36-39 

38 

102 

157 

32-35 

34 

37 

55 

28-31 

30 

15 

18 

24-27 

26 

3 

3 

A  casual  inspection  of  the  accompanying  graph,  of  the 
frequency  distribution  of  the  "raw  scores",  Figure  8,  as  cur¬ 
rently  obtained  reveals  a  figure  in  which  over  one-sixth  of 
the  frequency  population  falls  within  one  class  interval  near 
the  middle.  The  surrounding  class  intervals  seem  large  also. 
It  is  this  situation  apparently  which  caused  the  committee  to 
suspect  abnormal  crowding  near  the  middle  of  the  range e 

However,  a  mathematical  scrutiny  of  the  figure  shows 
that  it  is  not  the  least  leptokurtic.1  In  fact  its  kurtosis2 
is  practically  zero.  Neither  is  there  any  sign  of  skewness. 
The  distribution  is  almost  symmetrical  as  will  be  seen  readily 

1Leptokurtic  -  sharply  peaked.  2  Kurt os is  -  measure  of 

peakedness . 
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600 


Fig.  8.  -  Frequency  Distribution  of  Raw  Scores, 
English  5,  1943. 
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when  one  observes  that  the  median,  mean  and  approximate  mode 
are  54.5,  54.2  and  54  respectively.  A  normal  curve,1  when 
superimposed  coincides  almost  so  completely  that  it  is  dif¬ 
ficult  to  distinguish  them  on  the  diagram. 

Assuming  that  the  abilities  in  English  measured  by  the 
English  3  Examination  are  normally  distributed,  the  opinion, 
then,  that  there  is  abnormal  crowding  near  the  middle  of  the 
range  is  unsubstantiated.  However,  it  does  appear  that  the 
ranged,  which  extends  from  27  to  87,  that  is,  only  61  percent 
of  the  possible  range,  is  quite  inadequate  for  the  purpose  of 
discriminate  testing.  This  will  be  investigated  presently. 

At  the  moment,  however,  it  is  desirable  to  examine  the  fre¬ 
quencies  of  the  component  scores  that  go  to  make  up  the  ’’raw 
score”. 


iThe  normal  curves  throughout  this  work  were  constructed 
from  the  Tables  of  Ordinates  of  the  Normal  Probability  Curve, 
as  given  in  Walker;  Statistical  Methods,  1943,  Holt  &  Co. 

c 

The  tern  ”range”  is  used  throughout  this  work  to  indicate 
the  number  of  score  intervals  between  the  highest  and  lowest 
scores.  Thus  inadequacy  of  range”  is  to  be  considered  as  syn¬ 
onymous  with  ’’paucity  of  intervals”.  The  difference  lies  in 
the  method  of  approach  to  the  remedy  for  the  situation;  in  the 
former,  ways  and  means  are  sought  to  widen  the  range  while  in.' 
the  latter,  the  remedy  is  sought  in  an  increase  of  intervals  by 
means  of  introducing  fractions  of  marks.  Both  seek  to  increase 
the  number  of  intervals  in  the  range.  The  investigator  has 
adopted  the  former,  since  it  is  thought  that  in  a  subject  like 
English  the  discrimination  limen  is  large  and  not  well  adapted 
to  fractional  marking. 
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In  Table  V  which  follows,  a  tabulation  of  the  candidates* 
marks  on  Part  I  A  is  given.  This  is  followed  in  Table  VI  by 
a  tabulation  of  the  **Score-A*T  marks.  "Score- A"  is  computed  for 
each  candidate  by  disregarding  the  marks  assigned  by  the  teach¬ 
er  in  Part  IA;  that  is,  the  mark  for  Part  I  B  (maximum  100) 
is  added  to  the  total  language  mark  (maximum  100)  and  the  sum 
divided  by  2.  This  gives  a  score  on  a  percentage  basis  in¬ 
dependent  of  the  teacher’s  score. 

The  tables  are  followed  by  the  graphs  of  these  frequency' 
distributions. 


TABLE  V 

ENGLISH  3:  DISTRIBUTION  OF  PART  I  A  SCORE,  1943 


Score 

Frequency 

Cumulative  F. 

100 

1 

3286# 

95 

8 

3285 

90 

25 

3277 

85 

96 

3252 

80 

158 

3156 

75 

297 

2998 

70 

397 

2701 

65 

499 

2304 

60 

563 

1805 

55 

436 

1242 

50 

378 

806 

45 

202 

428 

40 

137 

226 

35 

54 

89 

30 

19 

35 

25 

11 

16 

20 

5 

5 

#A  few  candidates  did  not  receive  a  Part  I  A  mark. 
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Mean. 


.  9.  -  Frequency  Distribution  of  Part  I- A  Score, 
L943. 
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TABLE  VI 


ENGLISH  3:  DISTRIBUTION  OF  "SCORE- A",  1943 


Score 

Mid-Pt. 

Frequency 

Cumulative  F. 

80-83 

82 

4 

3320 

76-79 

78 

15 

3316 

72-75 

74 

22 

3301 

68-71 

70 

74 

3279 

64-67 

66 

155 

3205 

60-63 

62 

244 

3050 

56-59 

58 

359 

2806 

52-55 

54 

479 

2447 

48-51 

50 

553 

1968 

44-47 

46 

515 

1415 

40-43 

42 

381 

900 

36-39 

38 

294 

519 

32-35 

34 

138 

225 

28-31 

30 

59 

87 

24-27 

26 

22 

28 

20-23 

22 

5 

6 

16-19 

1 

18 

1 

1 

550 

500 

450 

400 

350 

300 

250 

200 

150 

100 

50 

0 

I 

*  « 
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Score-A _ 

Normal  Curve 


10.  *  Frequency  Distribution  of  Score  -  A, 
1943. 
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TABLE  VII 

THE  CUMULATIVE  FREQUENCY  DISTRIBUTIONS 
OF  THE  RAV  SCORE  AND  THE  SCORE-A 


Score 

Departmental  Raw  Score 

Competitive  Part  of  Exam 
i.e.  "Score- A" 

F 

Cum.F. 

Cum.% 

F 

Cum.F, 

Cum.% 

84-87 

2 

3316 

100 $ 

0 

80-83 

5 

3314 

99.9$ 

4 

3320 

100$ 

76-79 

26 

3309 

99.7$ 

15 

3316 

99.8$ 

72-75 

83 

3283 

99.0$ 

22 

3301 

99.4  $ 

68-71 

132 

3200 

96.5$ 

74 

3279 

98.7$ 

64-67 

258 

3068 

92.5$ 

155 

3205 

96.5$ 

60-63 

436 

2810 

84.7$ 

244 

3050 

91.8$ 

56-59 

489 

2374 

71.5$ 

359 

2806 

84.5$ 

52-55 

586 

1885 

56.8$ 

479 

2447 

73.7$ 

48-51 

521 

1299 

39.1$ 

553 

1968 

59.2$ 

44-47 

385 

778 

23.4 $ 

515 

1415 

42.6$ 

40-43 

236 

393 

11.8$ 

381 

900 

27.1$ 

36-39 

102 

157 

4.7$ 

294 

519 

15.6$ 

32-35 

37 

55 

1.6$ 

138 

225 

6.8  $ 

28-31 

15 

18 

.5$ 

59 

87 

2.6$ 

24-27 

3 

3 

.009$ 

22 

28 

.08$ 

20-  3 

5 

6 

.01$ 

16-19 

1 

1 

.003$ 
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Scores  in  Percentages 

Fig.  11.  -  Ogives  of  Raw  Score  and  Score-A,  English  3, 

1943. 
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From  Tables  IV,  V,  VI  and  VII  and  Figures  8,9,10  and 
11,  the  following  facts  may  be  deduced. 


TABLE  VIII 

FACTS  CONCERNING  THE  FREQUENCY  DISTRIBUTION 
OF  ENGLISH  3,  1943 


Score-A 

Part  I  A  Score 

Raw  Score 

Range 

17-83 

67 

20-100 

81 

27.87 

61 

Median 

49.8 

58.9 

(Actually  60) 

54.5 

Approx* 

Mode 

50 

60 

|  54 

: 

Mean 

49.5 

61 

54.2 

Standard 

Deviation 

9.7 

12.3 

9.15 

Interquartile  j 
Range 

1 

44-56.6  1 
12.6 

50.2-67 

16.8  >  I 

■ 

£ 

-  -  -  - . -  -  -  -  -  ■  -  -  * 

48.5-61.3 

12.8 

P90-P10 

36-61 

J 

42.5-74 

41-65 

80% 

25 

31.5 

24 

Again,  we  notice  that  the  frequency  distribution  of 
the  Part  I  A  Score  r ©presented  in  Figure  9  gives  an  almost 
syrifetrical  curve,  though  not  as  completely  as  that  of  the 
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nraw  score **  in  Figure  8.  In  the  Part  I  A  Score,  the  mean, 
median  and  approximate  mode  are  in  slightly  different  pos¬ 
itions;  the  values  of  these  are:  mean  61,  mode  60  and  median 
58.9.  although  this  latter  is  an  actual  60.  In  superimposing 
a  normal  curve,  we  note  that  the  class  frequency  at  the  mean 
of  the  normal  distribution  is  534  as  compared  with  550  in 
the  actual  distribution;  the  actual  frequency  polygon  is  then 
very  slightly  more  peaked  than  the  normal.  Apart  from  this 
the  distribution  practically  coincides  with  the  normal,  This 
is  all  the  more  remarkable  in  view  of  the  fact  that  the 
teachers1  marks  are  given  only  in  multiples  of  five. 

It  is  quite  evident  then  that  the  teachers  have 
gauged  rather  accurately  the  work  of  the  students  as  a  group, 
that  is^ if  the  normal  curve  is  a  reliable  index  of  this. 

Consider  now  the  distribution  of  the  Score-A  as  re¬ 
presented  by  the  frequency  polygon  in  Figure  10.  The  Score- 
A,  as  previously  pointed  out,  is  computed  by  leaving  the 
teachers*  marks,  Part  I  A,  out  of  the  calculation.  It  re¬ 
presents  the  percentage  obtained  by  the  student  on  the  De¬ 
partment’s  written  competitive  tests  only.  Despite  the  fact 
that  the  omission  of  the  teacher’s  mark  may  change  the  weigh¬ 
ting  of  the  other  marks,  yet  the  Score-A  represents  what 
would  prevail  if  it  should  be  decided  to  discontinue  the 
Part  I  A  mark. 

The  figure  presented  in  the  graph  is  quite  similar  in 
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contour  to  that  of  the  Part  I  A  Score  shown  in  Figure  9. 

Here  again  we  notice  that  the  mean,  median  and  mode  are  in 
almost  identically  the  same  position;  the  values  are  49.5, 

49.8,  and  50  respectively.  Likewise,  the  frequency  at  the 
mean  for  the  normal  distribution  is  slightly  less  than  the 
actual.  Again,  we  observe  how  the  actual  distribution  and 
the  normal  distribution  practically  coincide. 

Again  it  is  found  that,  if  normality  of  distribution 
is  accepted  as  an  indication  of  the  reliability  of  a  test 
in  English,  the  score  on  the  Departments  written  tests,  like 
the  marks  assigned  by  the  teachers,  is  also  an  accurate  meas¬ 
ure  of  the  pupil’s  abilities  in  the  subject. 

The  next  step  is  to  study  the  effects  of  the  two  dis¬ 
tributions  on  one  another  when  brought  together  in  the  compilation 
of  the  raw  score.  It  follows  quite  naturally  that,  since  tie 
Score-A  distribution  is  a  series  of  values  pitched  at  a  lower 
degree  in  general  than  the  series  comprised  by  the  Part  I  A 
distribution,  the  mutual  effect  would  be  a  moderating  one. 

An  examination  of  figure  12  shows  precisely  this  effect.  The 
distribution  of  the  Score-A  is  centred  on  the  score  50  line; 
that  of  the  Part  I  A  is  centred  on  the  score  60  line;  the 
resultant  distribution,  nraw  score*’,  is  situated  almost  mid¬ 
way  between  theip,  that  is,  it  is  centred  on  the  score  54  line. 

The  apparent  effect  then  of  the  Part  I  A  Score  op  the  Score-A 
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Fig.  12.  -  Comparative  Study  of  English  3  Distributions, 
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has  been  simple  to  raise  the  mean  about  five  points  or  in  other 
words,  to  transplant  the  distribution  without  much  distortion 
five  points  higher  on  the  scale.  This  is  really  of  no  consequence 
since  the  pass  mark  is  not  a  fixed  point  on  the  scale  but 
rather  a  value  determined  after  the  marking  which  will  cause 
a  certain  percentage  of  the  candidates  to  fail. 

This  effect  is  possibly  more  clearly  shown  in  Figure 
11  where  the  continuous  ogive  at  the  left  represents  the 
"Score-A"  and  the  dotted  ogive  represents  the  "Raw  Score". 

These  two  ogives  describe  the  cumulative  frequencies  both 
numerically  and  by  percentages  for  the  two  distributions. 

It  will  be  seen  that  the  two  curves  "parallel"  one  another 
over  the  greater  part  of  the  range  until  the  extreme  limit 
is  reached.  In  effect  this  means  that  on  the  average  the 
teachers1  marks  were  such  as  to  virtually  raise  the  score 
on  the  written  tests  by  approximately  five  percent.  Since 
the  teacher’s  mark  accounts  for  only  one-quarter  of  the  raw 
score  mark,  then  five  marks  on  the  raw  score  is  equivalent 
to  twenty  marks  on  the  teacher’s  score.  Hence,  it  follows 
that  on  the  basis  of  the  1943  examination  only,  either  the 
teachers’  marks  were  twenty  percent  "too  high"  or  the  Depart¬ 
ment’s  marks  were  correspondingly  "too  low",  if  it  is  de¬ 
sired  to  make  them  mutually  ineffective. 

(It  might  be  noted  here,  that  whereas  the  preceding 
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argument  considers  the  situation  from  the  view-point,  "ef¬ 
fect  of  teachers*  marks  on  the  Department *s  marks  in 
producing  the  raw  score",  it  might  also  be  considered  from 
the  view-point , "effect  of  the  Department’s  marks  on  the 
teacher’s  marks  in  producing  the  raw  score".  In  other 
words  it  is  not  to  be  assumed  that  one  is  correct  and  the 
other  incorrect.  ) 

It  is  not  necessary  nor  is  It  even  to  be  expected, 
except  according  to  the  laws  of  chance,  that  the  two  dis¬ 
tributions  would  coincide.  This  could  only  happen  on  con¬ 
dition  that  the  many  variable  factors  remained  constant  and 
that  the  rating  scales  of  each  were  tuned  to  each  other. 

The  purpose  served  by  the  two  types  is  that  one  might  com¬ 
pensate  for  the  defect  in  the  other.  If  the  two  were  cons¬ 
tantly  in  unison,  one  or  the  other  could  be  discarded. 

The  mutual  effect,  then,  of  the  two  distributions  - 
Part  I  A  and  Score-A  -  on  one  another  may  be  summed  as  fol¬ 
lows.  Each  distribution  is  by  itself  an  almost  normal 
distribution  and  when,  by  the  method  of  computation  now 
prevailing,  they  are  brought  together  to  give  the  raw  score, 
the  resulting  distribution  is  likewise  an  almost  normal 
distribution.  Hence,  the  surmise,  which  prevailed  prior 
to  the  investigation,  that  the  Part  I  A  Score  might  be 
exercising  a  distorting  influence  on  the  results  of  the 
written  tests,  is  not  substantiated.  Further,  the  effect  of 
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combining  the  two  distributions  has  been  to  give  a  resultant 
distribution  (the  distribution  of  the  raw  score)  centred 
about  a  mean  point  higher  in  the  scale.  That  is,  the  Part  I 
A  distribution  with  mean  point  at  61  when  combined  with  Score 
-A  distribution  centred  about  mean  point  49.5,  produced  a  Raw 
Score  distribution  with  mean  point  at  54.2.  Speaking  non- 
technieally  we  may  say  that  the  effect  has  been  to  move  the 
distribution  up  the  scale  by  five  points. 

It  must  kept  in  mind,  however,  that  we  have  been 
dealing  with  "total  effects"  only,  that  is,  the  effects  in 
so  far  as  they  concern  the  group  as  a  whole.  Effects  on  the 
individual  scores  are  not  being  considered  at  this  point. 
These  will  be  dealt  with  in  a  later  chapter. 

The  Effect  of  Part  I  A  on  the  Pass  Mark 

The  pass-mark  in  Alberta  Departmental  Examinations  is 
not  a  fixed  score  in  the  sense  that  it  is  a  certain  percent¬ 
age  of  items  on  the  examination  paper  answered  correctly  re¬ 
gardless  of  all  other  factors.  The  following  method  is  used. 

The  answer  papers  are  evaluated  according  to  a  scale 
that  may  or  may  not  add  up  to  a  total  of  one  hundred . 1 


4*or  more  complete  details  on  determining  the  raw-score 
on  the  English  examination,  see  card,  Fig.  7,  Chapter  II* 
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Fig,  13.  -  Transmutation  Scale  for  Score-A. 
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The  results  are  tabulated  and  from  those  forming  the  lower 
end  of  the  distribution  a  p re -determined  percentage  of  the 
entire  number  of  candidates  are  declared  failures  In  other 
words,  the  pass  mark  is  the  score  at  a  certain  percentile 
of  the  distribution.  The  marks  of  the  scale  are  then  trans¬ 
muted  so  that  this  particular  score  becomes  40,  while  the 
lowest  mark  and  the  highest  mark  made  on  the  examination 
become  0  and  100  re spec  timely .  The  other  marks  are  then 
determined  from  the  transmutation  scale.  Figure  13  illustrates 
the  transmutation  scale  which  was  constructed  for  use  with 
the  Score-A  marks.  Note  that  the  highest  mark  below  40,  that 
is,  the  highest  mark  of  a  failing  student  is  a  predetermined 
37;  also  the  highest  mark  below  50  is  always  47;  the  reason 
is  obvious. 

On  the  1943  examination,  it  was  predetermined  to  fail 
15#  of  the  candidates.  Following  the  method  described  above, 
it  was  found  that  failing  15#  of  the  candidates  would  set 
the  pass  mark  according  to  raw-score  at  45  and  that  the  lowest 
mark  in  B  category  was  52.  When  the  same  procedure  was  fol¬ 
lowed  for  the  Score-A  results,  the  pass  mark  became  40  and 
the  lowest  mark  in  the  B  category,  46.  It  must  be  pointed  out 
here  that  although  15%  of  3320  is  498,  yet  through  unevenness 
of  distribution  over  the  class  interval,  only  461  candidates 
became  failures  according  to  the  raw-score  while  519  were 
included  when  calculations  were  made  on  the  Score-A  distri- 
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tribution.  This  makes  a  slight  difference. 

The  effect,  then,  of  the  Part  I  A  Score  on  the  Score- 
A  in  regard  to  the  pass  mark,  is  to  raise  the  pass  mark  from 
40  to  45  and  to  raise  B  Category  from  46  plus  to  52  plus. 

This  is  in  accordance  with  what  was  seen  previously  where  the 
effect  was  to  move  the  distribution  approximately  five  points 
higher  along  the  scale. 

The  Effect  on  the  Kange 

While  range  is  not  a  very  effective  measure  of  the 
central  tendency  of  a  distribution,  yet  occasionally  it  is 
worth  considering.  Thus  an  extremely  small  range  might 
delight  us  if  we  were  interested  in  seeing  how  little  a  given 
phenomenon  deviated  from  a  standard.  On  the  other  hand  where 
we  wish  to  discriminate  between  several  degrees  of  effect- 
iveness,  we  prefer  a  wide  range  provided  the  class  intervals 
are  well  filled  out.  Thus  it  is  in  the  case  under  consider¬ 
ation;  we  want  to  distinguish  several  degrees  of  ability  in 
English.  Hence,  it  is  desirable  to  have  a  fairly  wide  range, 
so  that  the  frequency  in  any  class  interval  will  not  be  greater 
than,  let  us  say,  ten  percent  of  the  total. 

In  the  distributions  under  consideration,  out  of  a 
possible  range  in  each  case,  Score-A  has  a  range  of  17  to  83 
or  a  total  of  67,  Part  I  A  Score  has  a  range  of  20  to  100  or 
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a  total  of  81,  while  the  Raw  Score  has  a  range  of  27  to  87 
or  a  total  of  61.  Thus,  it  is  seen  that,  when  united  to¬ 
gether,  the  effect  of  the  Part  I  A  Score,  with  its  wide 
range  of  81,  on  the  Score-A's  range  of  67  is  to  reduce  it  to 
61.  (This  may  seem  rather  strange  but  a  study  of  the  in¬ 
dividual  scores  constituting  the  extremities  of  the  ranges 
shows  that  the  effect  has  been  to  draw  these  within  the 
deeper  part  of  the  distribution;  the  individuals  constituting 
the  extremities  were  not  the  same  in  each  distribution) 

It  is  noted  also  that  the  lower  end  of  the  range  was 
raised  from  17  to  27  while  the  upper  end  was  increased  from 
83  to  only  87.  That  the  lower  end  should  receive  a  greater 
increase  than  the  upper  end  is  accounted  for  possibly  by  rea¬ 
son  of  the  fact  that  the  possible  difference  in  scores  for 
the  two  distributions  is  greater  for  the  lower  end  of  the 
range  than  the  upper.  Thus,  for  example,  a  candidate  who 
received  17  marks  on  the  Score -A  could  receive  any  one  of 
the  marks  of  Part  I  A  from  20  to  100,  all  of  which  are  greater 
whereas  the  candidate  who  received  83  on  the  Score-A  can 
receive  only  marks  of  85  to  100  which  are  greater  but  might 
also  receive  any  one  of  the  marks  of  20  to  80  which  are  less. 

The  effect  does  not  seem  greatly  significant  part¬ 
icularly  in  view  of  the  fact  that  it  has  already  been  shown 
that  the  means  of  the  distributions  have  been  raised  by  ap¬ 
proximately  five  points.  Since  it  is  customary  to  associate 
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median  rather  than  mean  with  the  range,  it  is  just  as  well 
to  state  here  for  the  sake  of  the  record,  that  the  effect 
of  the  Part  I  A  Score  on  the  median  of  the  Score-A  dis¬ 
tribution  has  been  to  raise  it  by  approximately  five  points, 
that  is,  from  49.8  to  54.5.  This  follows  from  previous 
statements  to  the  effect  that  the  means  and  medians  of  the 
three  distributions  are  practically  coincident. 

A  reference  to  the  other  ranges  will  prove  rather 
interesting.  Thus,  for  the  Score-A  distribution,  the  inter¬ 
quartile  range, or  the  middle  50 $  of  the  scores  has  a  range 
of  only  12.6  marks;  the  plus  and  minus  sigma  range,  or 
middle  68$  of  the  scores,  has  a  range  of  only  19.4  marks; 
while  the  PgQ  -  P^0  range  or  middle  80$  of  the  scores  has 
a  range  of  only  25  marks.  The  distribution  of  the  Haw  Score 
has  ranges  so  similar  to  the  Score-A  distribution  that  the 
effect  of  the  Part  I  A  distribution  on  its  make-up  may  be 
considered  almost  negligible.  These  ranges  are  as  follows: 
Interquartile  range,  12.8;  plus  and  minus  sigma  range,  18.3; 
and  the  PQq  -  P^q  range,  24. 

The  investigation  substantiates  quite  fully  the  fact 
that  there  is  an  inadequate  range! in  the  marks.  It  shows 
also  quite  clearly,  that  since  the  ranges  of  the  Raw  Score 
and  the  Score-A  are  equally  narrow,  that  the  effect  is  in  no 
way  attributable  to  the  influence  of  the  Part  I  A  marks. 

By  way  of  illustration  of  the  effect  of  a  wide  range 

^See  footnote,  page  20 
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TABLE  IX 


A  COMPARISON  OF  THE  IDEAL  FREQUENCY  DISTRIBUTION 
AND  THE  ACTUAL  DISTRIBUTION  OF  THE  RAW  SCORE 


Ideal 

Actual 

Score 

Mid.  Pt. 

F# 

F 

96-99 

98 

4 

92-95 

94 

9 

88-91 

90 

17 

84-87 

86 

30 

2 

80-83 

82 

49 

5 

76-79 

78 

76 

26 

72-75 

74 

110 

83 

68-71 

70 

156 

132 

64-67 

66 

201 

258 

60-63 

62 

247 

436 

56-59 

58 

286 

489 

52-55 

54 

313 

586 

48-51 

50 

322 

521 

44-47 

46 

313 

385 

40-43 

42 

286 

236 

36-39 

38 

247 

102 

32-35 

34 

201 

37 

28-31 

30 

156 

15 

24-27 

26 

111 

3 

20-23 

22 

76 

j 

16-19 

18 

49 

12-15 

14 

30 

8-11 

10 

17 

4-7 

6 

9 

0-3 

2 

| 

4 

there  is  here  presented,  a  tabulation  of  the  frequencies  for 
the  same  number  of  candidates  the  distribution  remaining 
normal,  but  with  the  range  increased  to  a  maximum  of  100. 

In  the  table,  the  Raw  Score  frequencies  have  been  placed  in 
an  added  column,  for  the  sake  of  comparison.  The  graph, 

1  E  -  Ni  multiplied  by  respective  ordinate  values  of  x 
from^the  Table  of  Ordinates  of  a  Unit  Normal  Curve. ^ 


Humber  of  Candidates 


40 


Fig.  14.  -  Frequency  Curves  of  Ideal  Distribution  and 
Actual  Distribution. 
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(Figure  14},  shows  the  two  distributions  on  the  same  scale. 

It  is  quite  clear  how  much  more  accurate  the  subdivision 
becomes  when  the  class  frequencies  are  small  and  how  much 
more  meaningful  is  the  distinction  between  a  Grade  A  and 
a  Grade  D.  This  is,  it  seems,  what  the  committee  had  in 
mind,  when  they  complained  of  the  crowding  near  the  middle 
of  the  range. 

It  should  be  clearly  understood  at  this  point  that 
this  desirable  condition  cannot  be  brought  about  by  the 
method  of  transmutation  of  scores  currently  used  by  the 
Department.  This  method  of  transmutation  changes  the  range  to 
a  -  0  -  100  range  but  it  leaves  the  frequencies  the  same. 

This  is  illustrated  in  the  following  simplified  example 
where  the  marks  of  the  3000  candidates  ranged  from  50  to  60. 
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It  will  be  seen  that  the  effect  of  the  transmutation 
is  merely  to  separate  the  scores  with  blank  space.  It  does 
not  change  the  size  of  the  frequencies.  Some  of  the  evils 
of  a  narrow  range  are  also  illustrated.  In  the  example,  It 
is  seen  that  more  students  (over  1/3  of  the  total)  received 
H  standing  than  any  other,  because  it  was  impossible  to 
make  finer  subdivisions.  Further,  the  difference  In  marks 
between  those  who  scored  E  and  those  who  failed  is  so  small 
that  it  might  in  many  cases  be  the  result  of  testing  error. 

The  remedy  for  this  condition  seems  to  be  in  a  stand¬ 
ardisation  of  the  examination  paper  before  the  marking  is 
done,  that  is,  the  various  items  which  constitute  the  paper 
should  be  properly  weighted.  This  can  be  done  by  selecting 
a  sample  of  about  200  papers  and  marking  the  answers  right  and 
wrong,  or  in  the  case  of  judgments  or  opinions  by  evaluating 
the  answers  as  A,B,C  or  D.  A  frequency  curve  of  correctness 
is  then  calculated  and  values  are  assigned  to  the  various  answers 
on  the  basis  of  their  respective  frequencies.  The  committee 
should  be  able  to  work  out  the  details  of  this  method.  Before 
the  final  computation  is  made,  It  might  be  necessary  to  norm¬ 
alize  the  Part  I  A  Score,  If  it  Is  not  already  normally  dis¬ 
tributed  ,  and  to  scale  it  according  to  the  scale  of  the  writ¬ 
ten  tests.  The  combining  , of  the,, Part  I  A  Score  with  the  marks 
for  the  written  portion  should  be  the  last  step  In  the  cal¬ 
culation.  With  this  procedure,  an  almost  normal  distribution 
over  a  complete  range  will  be  obtained. 
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CHAPTER  TV 


THE  EFFECT  OF  THE  PART  I  A  MARK 
ON  THE  INDIVIDUAL  SCORES 

In  the  preceding  chapter  we  studied  the  effects  of 
the  Part  I  A  mark  on  the  distribution  of  the  final  score 
considered  in  its  totality.  It  was  seen  that  there  was 
no  distorting  effect  of  any  consequence  produced  on  the 
Score-A  distribution  in  forming  the  Raw  Score  distribution 
from  it.  It  does  not  necessarily  follow  that  each  individual 
in  the  distribution  was  effected  in  precisely  the  same  man¬ 
ner  as  the  entire  distribution.  Indeed,  the  effect  on  the 
individual  member  of  the  distribution  is  a  very  important 
aspect  of  the  study.  It  is  this  effect  which  is  to  be  the 
subject  of  the  investigation  in  this  chapter. 

Effect  on  the  Individual  Marks 

For  each  candidate  the  mark  which  he  would  have 
scored  if  Part  I  A  had  not  been  part  of  the  examination  has 
been  computed  and  is  referred  to  as  his  Score-A  mark.  To 
compute  this,  the  mark  on  his  literature  paper  (100%)  is  ad¬ 
ded  to  the  total  marks  for  language  (100%)  and  the  sum  divided 
by  2.  This  mark  is  to  be  compared  with  his  final  score  as 
currently  obtained.  For  this,  the  Department’s  Raw  Scores  are 
used.  A  Raw  Score  is  computed  by  adding  the  Part  I  A  mark  to 
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the  Part  I  B  mark  and  dividing  by  2;  this  gives  the  mark  for 
literature;  the  literature  mark  and  the  total  language  mark, 
each  100%,  are  added  together  and  divided  by  2.  This  gives 
the  Raw  Score  for  English  based  on  100%.  To  be  clear  on  the 
point  the  "Score-A"  mark  excludes  the  Part  I  A  mark  whereas 
the  Raw  Score  includes  it. 

In  the  accompanying  tables  is  shown  the  effect  of  the 
Part  I  A  mark  on  the  Score-A  mark,  in  producing  a  Raw  Score. 
This  is  done  by  pointing  out  for  each  Score -A  mark,  the 
amount  of  change  expressed  in  marks  that  has  been  brought 
about  by  including  the  Part  I  A  mark  in  the  ‘calculation.  In 
Table  X  this  has  been  done  for  the  entire  group  of  candidates. 
Tables  XI  and  XII  report  the  findings  separately  for  rural 
and  urban  candidates. 

Thus,  on  the  first  line  of  the  table,  one  and  only 
one,  candidate  would  have  received  seventeen  marks  if  the 
Part  I  A  marks  had  not  entered  into  the  calculation.  From 
the  table  we  learn,  as  we  go  across  the  first  line,  that  one 
candidate  received  a  Raw  Score  which  was  10  marks  in  excess  of 
his  Sc ore -A  mark.  Thus  the  Raw  Score  of  this  candidate  was 
increased  from  17  to  27  by  including  the  Part  I  A  mark  in  the 
calculation. 

In  the  table  the  column  headings  indicate  differences 
between  Raw  Score  and  Score-A,  that  is,  they  are  the  deviations 
of  the  Raw  Score  from  the  Score-A.  If  the  Score-A  is  increased 
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the  deviation  is  considered  positive,  if  decreased  it  is 
considered  negative.  If  no  change  occurs  it  will  be  listed 
under  the  0  heading.  The  numbers  along  the  left  hand  mar¬ 
gin  are  the  Score- A  marks  which  will  be  considered  in  each 
row.  The  numbers  in  the  main  body  of  the  table  are  the 
numbers  of  candidates  whose  marks  were  effected  as  indicated 
in  the  column  heading. 

For  a  further  illustration,  let  us  consider  a  Score-A 
of  £8.  Going  down  the  left  hand  margin  to  28  and  then  reading 
across  the  row  we  find  one  candidates  score  remained  un¬ 
changed,  one  received  an  increase  of  two, another  received  an 
increase  of  five,  two  received  an  increase  of  six,  three  re¬ 
ceived  an  increase  of  seven,  two  received  an  increase  of  eight, 
and  finally  one  received  an  increase  of  nine.  This  means  that 
of  the  eleven  candidates  whose  written  work  was  evaluated  at 
28$  by  the  examiners,  one  received  a  raw  score  of  28,  one  30, 
one  33,  two  34,  three  35,  two  36  and  finally  one  received  37, 
by  reason  of  the  influence  of  their  teachers*  marks.  This 
range  of  28  to  37  when  the  marks  are  transmuted  becomes  a  range 
of  2-22  or  a  total  range  of  21. 

For  a  further  illustratioh  let  us  select  a  row  with 
an  extremely  wide  divergence  -  for  example,  the  last  row  on 
the  first  page  of  the  table,  Score-A  52.  Here,  we  observe 
that  the  deviations  range  from  -6  to  16;  This  means  that 
although  the  written  papers  of  the  122  candidates  were  all 
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Deviations  of  the  Candidates*  Raw  Score  from  the  Corresponding  Score-A 

Fig.  15.  -  Histogram,  showing  Deviation  of  Raw  Scores,  from  nScore-AM  individually  for 
Jural,  Urban,  and  Total  Candidates. 
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valued  at  52,  their  Raw  Scores,  through,  the  effect  of  the  Part 
I  A  mark,  ranged  all  the  way  from  46  to  68  for  a  total  range  of 
23  marks;  these,  when  converted  to  transmuted  scores,  become 
a  range  of  41-73  for  a  total  range  of  53  marks.  It  is  true 
that  this  happens  to  be  taking  a  sample  at  the  point  where 
the  divergence  is  greatest.  Yet,  nevertheless,  the  fact 
that  it  is  not  an  isolated  case,  and  the  fact  that  such  can 
happen,  that  candidates  whose  written  work  is  apparently  of 
equal  calibre  can  receive  final  standings  which  range  from 
the  lowest  Grade  C’s  to  the  highest  Grade  A*s,  makes  one 
question  the  validity  of  the  test.  Can  such  an  examination 
be  a  reliable  test  of  ability?  Mere  the  candidates*  writ¬ 
ten  papers  really  of  equal  value?  Were  the  teachers  trying  to  pass 
the  students  in  spite  of  the  written  tests?  Or,  Is  the  result 
genuine  and  the  divergence  really  due  to  the  operation  of 
many  chance  factors? 

These  are  questions  which  prove  interesting.  A 
close  study  of  the  situation  leads  the  investigator  to  ac¬ 
cept  the  view  that  the  results  are  genuine  attempts  at  ac¬ 
curacy  and  that  the  deviations  are  due  to  the  operation  of 
a  great  many  extraneous  factors,  including  the  human  element 
involved  in  the  evaluation  of  the  papers  and  In  the  evalua¬ 
tion  of  the  candidate's  work  during  the  year.  That  the  de¬ 
viations  are  due  to  chance  rather  than  concerted  effort  is 
substantiated  by  a  consideration  of  the  normality  of  the 
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distribution  of  the  deviations  as  shown  by  the  histogram 
in  Figure  15.  The  mean  of  the  deviations  as  calculated 
from  the  totals  of  Table  X  is  only  4.7#  (the  same  as  was 
shown  in  the  last  chapter  when  dealing  with  the  change  of 
position  of  the  mean) .  The  indication  seems  to  be  that  the 
teachers T  marks  for  the  candidates  are  based  on  a  higher 
point  of  reference  than  that  of  the  Department.  This  ac¬ 
counts  for  a  considerable  portion  of  the  deviations,  as  is 
evidenced  by  the  fact  that  the  correlation  of  the  Raw  Score 
and  the  Score- A  is  f .90,  a  very  high  correlation. 

This  difference  in  the  point  of  reference  does  af¬ 
fect  the  score  slightly  since  it  changes  the  relativity. 

If  the  teachers’  marks  are  pitched  £5%  higher  than  the  De¬ 
partment’s,  then  the  teachers*  actual  range  becomes  £5-100 
instead  of  £5-l£5  or  0-100.  The  upper  end  becomes  distorted 
because  the  total  is  limited  to  100.  To  correct  for  this 
error  it  might  be  necessary  to  transform  the  scores  of  Part 
I  A  by  a  process  similar  to  McCall’s  T-Scaling  device.1 
But,  Whether  the  advantage  derived  would  be  commensurate  with 
the  time  and  expense  involved,  is  quite  problematical. 

The  better  way  of  accomplishing  this  vould  be  to 

^Consult,  such  texts  on  Statistics  as,  Walker:  Statistical 
Methods,  page  181-4,  Holt  &  Co.  1945. 

McCall:  Measurement,  Macmillan  1939. 

4,'The  Mean  of  the  Deviations  of  Raw  Scores  from  Scores-A  for 
Urban  candidates  is  3.36;  for  Rural  candidates  5.34. 
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standardize  the  teachers’  evaluations.  Not  only  would  this 
correct  the  preceding  fault,  but  it  would  also  minimize  the 
effect  of  another  cause  of  the  deviations,  the  present  lack 
of  standards.  With  600  teachers,  each  evaluating  according 
to  his  own  standard,  there  is  bound  to  be  a  considerable 
amount  of  variation  with  consequent  unfairness  to  the  can¬ 
didates.  If  a  set  of  standard  values  were  set  up  to  guide 
the  teacher  in  forming  his  estimate,  there  would  be  a  great¬ 
er  probability  of  uniformity. 

Further,  if  the  suggestion  made  in  the  preceding 
chapter  in  regard  to  widening  the  range  were  carried  out, 
the  undesirable  aspect,  whereby  the  candidates  are  moved 
into  other  gradings,  would  be  considerably  moderated.  The 
dividing  lines  of  the  different  categories  would  be  farther 
apart,  and  consequently  25  marks,  the  raw-score  value  of 
Part  I  A,  added  to  any  Score-A  could  not  possibly  raise  a  score 
more  than  two  categories. 

When  the  range  is  not  very  wide*  the  Part  I  A  mark 
may  constitute  more  than  an  influence  of  25$  on  the  final 
transmuted  score.  Its  influence  is  25$  of  the  Raw  Score  but 
in  the  process  of  transmutation  this  value  is  changed.  Thus, 
in  the  1943  examina  t ion,  the  range  was  27-87,  a  total  of 
67$  of  the  possible  range.  Let  us  see  the  total  possible 
influence  of  the  Part  I  A  mark  on  the  two  end  scores  of  the 
range  by  making  the  calculations  with  Part  I  A  as  zero  and 


■’  !  0- 

.  x;- 

■ 

o  ;■  ■  •  -0  .  ■  &o 

■ 

. 


. 


. 


.  .  ^  ■  v  '  :  ■  ...  :■  '  / 

1 

• '  ,  ■  ■  ;  :  $  X '  0. 

. 

* 

' 


. 

' 

•  ‘  -■  '  - 

' 

, 

. 

t ;  '  •  /  vUi  ' 

:  viC., 


56 


1 _ ) 

c _ 


again  as  100  and  comparing  the  effects. 


Effect  on  Raw  Score  27 


With  Part  I  A  as  0 

Part  I  A  as  100  * 

Difference 
i.e.  Part  I  A 

Lit .  A  0 

100 

Lit.  B  50 

50 

Total  (Lit)  50 

150 

Divided  by  2 . 

25 

. 75 

Language  (A&B).... 

29 

. . 29 

Total  . 

54 

.  . . 104 

Divided  by  2  equals 

j 

RAW  SCORE  . . 

27 

. .  52 

25 

Transmuted  Score 

0 

.  50 

50 

GRADE  . 

D 

_ ......  B 

i   .  ... — i — — 

_ 

Effect  on  Raw  Score  87 


With  Part  I  A  as 

100 

Part  I  A  as  0 

Difference 
i.e.  Part  I  A 

Lit.  A  100 

0 

Lit.  B.  48 

48 

Total  (Lit)  ..  148 

•  •  • 

48 

Divided  by  2  . 

.  74 

.  24 

Language  (A&B)  .... 

. . 100 

Total  . .  . 

. .124 

RAW  SCORE  . 

.  ... _ _  62 

25 

TRANSMUTED  SCORE  . 

.100 

.  64 

36 

GRADE  . 

.  B 

•  i 


. 

. .  . 

. . 
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It  will  be  seen  that  in  each  of  the  Raw  Scores, the 
Part  I  A  has  a  maximum  value  of  25  marks,  but  that  when  the 
scores  are  transmuted  its  value  is  increased  in  the  one  case 
to  50  marks  and  in  the  other  to  36.  Note  that  the  Part  I  A 
has  a  maximum  value  of  25  marks  and  not  25$;  it  may  be  greater 
than  25$  of  the  Raw  Score  or  the  Transmuted  Score.  Thus, 
in  the  first  case,  it  is  25  out  of  52  or  48$;  in  the  second 
case,  it  is  25  out  of  87  or  almost  29$. 

Since  this  condition  is  an  outcome  of  the  device  of 
lengthening  an  inadequate  rqnge  by  transmutation,  it  should 
be  possible  to  avoid  it  either  by  arranging  factors  so  as  to 
produce  a  complete  range  or  by  transmuting  the  Part  I  A  and 
Score-A  separately  before  combining  them  to  form  the  final 
score. 

Effect  on  the  Individual  Grades 

Another  interesting  way  of  presenting  the  matter  com¬ 
prising  the  earlier  part  of  this  chapter  is  to  review  the 
effect  which  the  Part  I  A  mark  plays  in  the  allocation  of  the 
Grade  letter  of  the  individual. 

The  method  used  is  similar  to  the  one  used  at  the  be¬ 
ginning  of  the  chapter.  It  was  pointed  out  there  how  the 
"Score-A"  was  determined.  These  scores  were  then  sorted  into 
a  frequency  distribution.1 


!See  Table  VI* 
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2 

From  this,  using  the  same  method  that  is  used  by  the  De¬ 
partment,  a  transmutation  scale  (Figure  13)  was  made.  The 
transmuted  scores  of  Score-A,  were  then  converted  to  the 
Grade  Letters  H,A,B,C,D. 

The  candidates*  file  cards  were  sorted  into  five 
piles  according  to  the  Grades  assigned  by  the  Department 
on  the  basis  of  the  transmuted  value  of  the  Raw  Scores. 

The  number  of  candidates  falling  into  each  category  may  be 
read  from  the  column  at  the  left  in  Table  XIII.  For  pur¬ 
poses  of  comparison  these  have  been  further  subdivided  into 
Rural  and  Urban  as  well),  thus  the  square  at  the  upper  left 
hand  corner  states  that  37  rural  candidates  and  39  urban 
candidates,  a  total  of  76  altogether  received  Grade  H  stand¬ 
ing. 

Each  pile  of  cards  was  then  sorted  out  according  to 
the  Grades  determined  on  the  basis  of  the  transmutations  of 
Score-A.  The  totals  of  these  may  be  read  along  the  hori¬ 
zontal  rows.  Thus,  of  the  76  candidates  who  received  E  stand¬ 
ing  on  the  basis  of  the  total  Raw-Score  only  65  would  have 
received  the  same  standing  on  the  basis  of  the  Score-A; 
the  others  would  have  been  reduced  to  A  standing. 

The  changes  in  grading  which  would  be  made  in  the  De- 

2 

The  method  is  fully  described  in  Chapter  IV. 
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TABLE  XIII 

COMPARISON  OF  DEPARTMENT  *  S  GRADING 
WITH  GRADING  ON  SCORE-A 


Grades  from 

Transmuted  Sc ore- A 

H 

.  

A 

B 

C 

D 

Rural 

37 

28 

9 

Urban 

39 

37 

2 

Total 

76 

65 

11 

Rural 

258 

21 

196 

40 

1 

Urban 

269 

30 

219 

20 

Total 

527 

51 

415 

60 

1 

Rural 

805 

101 

611 

84 

9 

Urban 

633 

: 

139 

468 

25 

1 

Total 

1438 

240 

1079 

109 

10 

Rural 

480 

] 

1 

104 

259 

116 

Urban 

338 

137 

170 

31 

Total 

818 

1 

241 

429 

147 

Rural 

296 

1 

49 

246 

Urban 

165 

3 

46 

116 

Total 

461 

4 

95 

362 

Rural 

1876 

49 

307 

756 

393 

371 

Urban 

1444 

67 

360 

628 

241 

148 

Total 

3320 

116 

667 

1384 

634 

519 
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partment’s  Grade  A,  B,  C,  D,  candidates  if  the  Part  I  A 
mark  were  to  be  eliminated,  may  be  read  from  the  chart. 

The  last  row  gives  the  total  of  the  columns.  From  this, 
we  learn  that  by  disregarding  the  Part  I  A  mark,  116  instead 
of  76  would  have  received  H,  667  instead  of  527  would  have 
received  A,  1384  instead  of  1438  would  have  received  B, 

634  instead  of  818  would  have  received  C  and  519"  instead 
of  461^  would  have  received  D  category  or  failure. 

If  we  express  these  figures  as  percentages  based  on 
the  Score- A  figures,  the  above  statement  would  read  like  the 
following.  Of  all  the  candidates  who  would  have  received  an 
H  standing  on  the  written  tests  alone,  34,5$  received  a  low¬ 
er  standing  by  reason  of  the  teachers*  Part  I  A  marks;  si¬ 
milarly,  a  net  21$  of  the  Grade  A’s  were  downgraded.  At  the 
same  time,  11.1$  of  those  who  would  have  failed  received 
higher  standing.  As  a  result  of  the  down  grading  of  the  H's 
and  Afs,  and  the  upgrading  of  the  Dfs,  3,9$  more  received  B 
standing  and  29$  more  received  C  standing. 

Considering  the  changes  in  grading  for  rural  and  urban 
candidates  separately,  we  notice  similar  effects.  For  the 
rural  candidates,  24.5$  of  the  Grade  H  and  16$  of  the  Grade 
A  were  downgraded  and  20.2$  of  the  D*s  were  upgraded  as  a 

^The  difference  between  these  is  due  to  unevenness  of 
distribution.  Both  aimed  at  being  498,  that  is,  15$  of 
the  total. 
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result  of  the  teachers1  Part  I  A  marks;  Grades  B  and  C  were 
increased  by  6,5 $  and  £1.9$  respectively.  Of  the  candidates  from 
urban  schools,  41.8$  of  the  Grade  H  and  £5.3$  of  the  Grade  A*s 
received  lower  grading.  However,  in  contrast  to  the  rural 
candidates,  the  percentage  of  failures  was  also  increased  by 
11.5$.  The  increases  in  Grades  B  and  C  were  0.8$  and  4.0. £$ 
re spectively . 

Table  XTV  displays  the  change  in  grading  in  another  way. 
In  the  column  marked,  "Raw  Score"  we  find  in  the  R  rows  the 
percentages  of  rural  students  in  each  category  when  both  the 
written  tests  and  the  Part  I.  A  marks  are  combined;  similarly, 
the  U  rows  give  the  percentages  of  urban  candidates  in  each 
category  and  the  T  rows  indicate  what  percentage  of  all  the 
candidates  fall  into  each  grade.  For  example,  of  the  1876 
rural  candidates,  1.97$  of  this  number  are  grade  H,  13.7$ 
are  Grade  A,  and  so  on.  In  the  next  column,  marked  "Score- 
A",  is  given  the  percentages  of  each  group,  Rural,  Urban,  or 
Total,  which  would,  fall  into  each  grade,  on  the  basis  of  the 
written  tests  alone  without  the  Part  I  A  marks.  The  third 
column- marked  "Difference",  shows  the  extent  of  the  effect  of 
the  Part  I  A  marks,  in  terms  of  difference  in  percentage. 

(For  those  who  may  wish  to  do  so,  the  table  affords  an 
opportunity  of  comparing  the  results  of  the  rural  and  urban 
candidates  in  the  subject  for  this  particular  examination. 

Compare  adjacent  Rfs  and  Ufs  in  each  square). 
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TABLE  XIV 


PERCENTAGE  DISTRIBUTION  OF  GRADES  COMPUTED  ON  BASIS  OF 
TRANSMUTED  RAW-SCORES  .AND  TRANSMUTED  SCORES-A  FOR 
RURAL,  URBAN  AND  TOTAL  CANDIDATES.  ENGLISH  3,  1943 


Grade 

♦ 

S===3S3S=SSS2S - - - - 

Raw  Score 

r  = . 

Score-A 

Difference 

R 

1.97 

2.6 

-  .63 

H 

U 

2.7 

4.64 

-1.94 

T 

2.29 

3.5 

-1.21 

R 

13,7 

16.3 

-2.6 

A 

U 

18.63 

24.9 

-6.27 

T 

15,87 

20.1 

-4.23 

R 

42.9 

40.3 

42.6 

B 

U 

43.9 

43.5 

.4 

T 

43.3 

41.7 

1.6 

R 

25.6 

20.95 

4.65 

C 

U 

23.4 

16.7 

6.7 

T 

24.54 

19.1 

5.44 

R 

15,77 

19.77 

-4.00 

D 

U 

11.4 

10.25 

1.15 

T 

13.9 

15.6 

-1.7 

Rural  percentages  computed  on  a  basis  1876  candidates. 
Urban  percentages  computed  on  a  basis  1444  candidates. 
Total  percentages  computed  on  a  basis  3320  candidates. 
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The  effect  of  the  Part  I  A  Score  is  clarified  by  a 
study  of  these  charts.  Briefly,  the  effect  is  a  central¬ 
izing  one,  that  is,  the  frequencies  at  the  extremes  seem 
to  be  reduced  and  the  central  ones,  the  B's  and  C's,  are 
increased.  If  we  consider  the  Departments  distribution 
of  grades,  based  on  the  Raw  Scores,  which 'include  the  Part 
I  A  marks,  and  compare  it  with  the  distribution  of  grades 
based  oh  the  Scores-A,  which  exclude  it,  we  perceive  this 
centralizing  effect.  In  the  earlier  part  of  this  chapter 
we  stressed  the  effect  of  Part  I  A  in  raising  the  mean;  in 
the  last  chapter,  we  showed  its  effect  in  condensing  the 
range;  in  this  section  we  see  the  complete  picture  of  both 
these  effects. 
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CHAPTER  V 


OTHER  EFFECTS  AND  RELATIONSHIPS 

The  work  contained  in  this  chapter  is  not  a  necessary- 
part  of  the  major  topic  under  discussion.  The  writer’s  prin¬ 
cipal  reason  for  including  it  is  that  the  information  was 
readily  available  from  the  work  done  in  the  previous  chapters 
and  may  not  be  so  again  unless  it  is  specially  sought.  Further, 
the  information  given  here  does  have  a  rounding  out  effect  in 
so  as  far  as  it  may  supply  facts  dealing  with  the  reliability 
of  the  present  English  examination  frame-work. 

Relationships  of  the  Literature  A  (Part  I  A) 

The  purpose  of  investigating  the  possible  relationships 
of  Literature  A  was  done  with  a  view  to  determining  to  what 
extent  the  measure  is  reliable.  In  the  first  place  it  was 
felt  that  there  ought  to  be  a  very  high  correlation  between 
Literature  A  and  Literature  B.  Presumably  both  purport  to 
measure''  the  same  matter:  Part  A  is  an  evaluation  of  "the  stu¬ 
dent’s  work  in  the  reading  and  study  of  Poetry,  Drame  and  Es¬ 
says n  ;  Part  B  is  Ma  test  of  reading  comprehension  and  literary 
appreciation  on  ’  sight1  selections ".  Ostensibly  they  are 
quite  similar.  However,  it  is  found  that  the  correlation 
between  the  Teachers1  marks  in  Literature  A  and  the  Depart¬ 
ment’s,  'marks  in  Literature  B  is  only  .46,(P.E.  .0092)  which  while 
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being  a  substantial  correlation,  is  nevertheless  much  smal¬ 
ler  than  had  been  anticipated.  The  correlation  was  made 
by  using  the  results  of  the  entire  group  of  candidates,  and 
so,  it  is  not  due  to  any  error  of  sampling. 

Before  discussing  the  possible  causes  of  this  dif¬ 
ference  in  scoring,  let  us  consider  two  or  three  other  re¬ 
lationships.  The  correlations  for  these  were  made  from  a 
sample  of  examination  papers  obtained  from  the  Department. 

The  reliability  of  the  sample  may  be  determined  by 
a  comparison  of  the  cumulative  percentage  frequency  ogives 
shown  in  Figure  16.  The  means  and  medians  of  the  two  dis¬ 
tributions  are  almost  coincident.  The  mean  of  the  whole 
population  is  54.2  while  the  mean  for  the  sample  is  54.3; 
the  median  for  the  whole  population  is  54.5  while  that  for 
the  sample  is  54.  It  may  be  taken  then  that  the  sample  is 
representative  of  the  entire  group. 

Using  the  sample,  a  correlation  was  made  between  the 
Literature  A  and  the  total  mark  for  Social  Studies.  It  was 
thought  that  there  was  sufficient  similarity  between  the 
processes  involved  in  the  study  of  Literature,  and  that  of 
Social  Studies  that  there  should  be  found  an  adequate  cor¬ 
relation.  Both  processes  involve  ability  (1)  to  read  in¬ 
tellectual  material  with  comprehension,  (2)  to  pass  judg¬ 
ment  on  what  is  read,  (3)  to  appreciate  the  content  and 
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Fig,  16.  -  Cumulative  Percentage  Frequency  Ogives  of 
Raw  Score  Distribution  and  Sample  Distribution,  English  3,  1943. 


Number  of  Candidates 


67 


Scores  in  Percentages 


Fig.  17.  -  Frequency  Distribution  of  Sample  from 
English  3,  1943. 
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(4)  to  remember  and  reconstruct  both  the  ideas  which  have 
been  read  and  those  which  have  been  formed  during  reading. 
The  correlation  yielded  was  rather  low,  .35,(P.E.  .041), 

Another  relationship , that  between  the  Literature  A 
and  Question  5  of  the  Literature  Examination  paper,  was 
studied. 

Question  5  is  as  follows: 

T,The  reading  and  study  of  literature  should  help  a 
student  to  know  a  good  book  when  he  sees  one  -  to  dieting- 
uish,for  instance,  between  the  temporary  and  the  permanent, 
the  false  and  the  true. v 

With  the  above  statement  in  mind  write  an  essay  in 
which  you  discuss  some  of  the  qualities  which  enable  you 
to  recognize  a  piece  of  literature  as  good.  You  may  use 
illustrations  from  fiction,  poetry,  essays,  or  drama,  and 
you  need  not  confine  yourself  to  one  field...... 

The  very  nature  of  this  question  would  lead  one  to 
expect  that  there  would  be  a  very  close  relationship  be¬ 
tween  the  scores  for  this  question  and  the  marks  assigned 
to  Literature  A.  The  information  sought  in  this  question 
seems  to  be  precisely  what  the  teaching  and  study  of  liter¬ 
ature  aims  primarily  to  give.  It  would  seem  that  the  pos¬ 
session  of  this  information  by  the  student  would  be  the 
criterion  by  which  the  teacher  would  judge  "whether  or  not 
the  student’s  work  has  been  consistently  and  continuously 
satisfactory  throughout  the  year " » "L 


-^Programme  of  Studies  for  the  Eigh  School,  Regulations 
of  the  Department  of  Education,  1944,  page  28. 


.  "  aaori  u  ■  j-  hi  ;£■' :  a  :-*6>d 

.  .  .  .  .d-d,  ..o;  bahS:C:l::  r±c  IC'B.lhh.Zi^o  oxiT 

.  .  ■  ■  :  ::  •  i  Jb  '  .  .  .  dr  - 

. 

, 

* 

a. :  .  i  r t-c ^  • 

. 

’ 

. 

, 

'  :  b d,  :  r:n  /ov  r  ■  J:*: r.:- D  ub  ,  -r  i  ;>v 

„froi  ;  -  <  « 


’ 

, 

/ 

. 

.  ,  ;  i  :■>  ■;  '  '■  -  .  y  l  dd  x""'  r.~ ; 


;  \  b-j iv  rce-  e>ic!  d©  Mw  \p  •aol'i®'$X%$ 


69  - - 


However,  once  again  the  correlation  was  less  than  was  anti¬ 
cipated.  In  this  relationship,  it  reached  the  almost  negligible 
low  of  .18,  (P.E.  .039). 

Between  the  Literature  A  and  the  average  for  the  De¬ 
partment’s  written  tests  in  English,  that  is,  the  Score-A, 
which  includes  both  literature  and  language,  where,  on  ac¬ 
count  of  the  many  factors  involved,  one  would  hardly  expect 
to  find  a  close  relationship,  there  exists  the  rather  substantial 
correlation  of  .51,  (P.E.  ,029). 

What  are  we  to  think  of  these  relationships?  It  is  quite 
apparent  that  no  conclusive  facts  are  revealed.  All  that  one 
can  do  is  surmise  the  reasons  for  the  disappointing  relation¬ 
ships  and  surmises  can  be  quite  far  from  the  truth. 

The  very  low  correlation  between  Literature  A  and  Question 
5  would  seem  to  indicate  that  the  factual  knowledge  of  literary 
criticism  studied  as  a  science  is  not,  and  is  not  considered 
by  the  teacher  to  be,  a  dominant  aspect  of  Grade  XII  English. 
Appreciation  of  literature  is  viewed  as  an  art  rather  than  a 
science,  something  to  be  felt  or  accepted  on  literary  faith, 
rather  than  to  be  reasoned  about  or  subjected  to  an  analysis. 

This  would  account  for  the  low  correlation;  the  pupil  might 
be  satisfactory  according  to  the  teacher’s  viewpoint  but 
when  subjected  to  the  fact  searching  of  the  examination,  he 
is  found  wanting.  Similarly  the  lack  of  a  high  degree  of 
correlation  between  Literature  A  and  Literature  B  may  be  part- 
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ly  explained  on  the  basis  of  lack  of  standards  in  judging. 

Many  more  factors  operating  over  a  period  of  time  enter  into 
the  formation  of  the  teacher’s  judgment,  whereas  the  Depart¬ 
ment’s  score  is  based  on  a  single  test  given  at  a  time  which 
may  not  have  been  opportune.  Estimates  of  value  are  often 
distorted  because  in  the  mind  of  the  evaluator  certain  factors 
may  receive  undue  stress  whereas  objective  evaluations  may 
be  revised  to  meet  the  criticisms  of  many  evaluators. 

Hence,  it  is  the  opinion  of  the  investigator  that  if 
definite  standards  for  evaluation  were  set  up,  a  greater  de¬ 
gree  of  agreement  might  be  attained  after  a  few  years. 

( 

Relationships  of  the  Language  Parts  of  the  Examination 

On  the  examination,  the  candidate’s  ability  to  write 
a  paragraph  of  prose  is  tested  on  three  separate  examination 
papers.  First,  Question  5  of  the  Literature  paper  is  evaluated 
independently  of  the  correctness  of  its  literary  content  for 
its  quality  as  good  prose;  this  is  marked  on  a  basis  of  30 
and  constitutes  what  is  referred  to  as  the  Language  A  score. 
Secondly,  Question  5  of  the  Language  paper  calls  for  the 
writing  of  a  paragraph  of  prose  on  one  of  the  topic  sentences 
selected  from  about  a  dozen  on  a  great  variety  of  subjects. 

,  a  $  1 1 

This  composition  is  marked  on  10  and  constitutes  a  part  of 
the  Language  B  marks.  Thirdly,  the  Social  Studies  paper  was 
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marked,  apart  from  its  Social  Studies  content,  for  its  ex¬ 
cellence  as  prose  writing.  This  is  marked  on  20  and  forms 
part  of  the  total  Social  Studies  mark.  Each  of  these  e- 
valuations  was  made  by  using  the  same  special  scoring  tech¬ 
nique. 

Here  we  have  the  marking  of  three  samples  of  the 
same  candidates1  writing  under  nearly  the  same  conditions. 

A  study  of  the  relationship  between  them  shows  a  correla¬ 
tion  of  .28  (P.E.  .037)  between  Language  A  and  Question  5  of 
the  Language  paper  and  a  correlation  of  only  .23  (P.E.  .044) 
between  Language  H  and  the  Language  of  the  Social  Studies  paper. 

Despite  the  fact  that  we  are  definitely  measuring 
similar  abilities  the  measurements  yield  only  a  very  low  cor¬ 
relation.  It  would  seem  that  when  the  candidates  are  writing 
Literature  or  Social  Studies,  their  minds  are  more  attentive, 
to  recalling  content  than  to  the  manner  of  expressing  their 
thoughts.  But  this  does  not  tell  the  whole  story.  Other¬ 
wise,  there  should  be  substantial  correlation  bet ween  Lan¬ 
guage  A  and  the  Language ■ of  the  Social  Studies  examination, 
since  both  are  measured  on  papers  that  serve  a  dual  purpose. 

An  inspection  of  the  marks  reveals  that  quite  generally 
the  marks  for  the  language  of  the  Social  Studies  paper 
were  higher  than  those  assigned  to  the  language  on  the 
Literature  paper.  Probably  the  difficulty  of  content  of  the 
answer  is  an  important  factor;  it  is  perceived  that  the  marks 
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for  the  literary  content  of  Question  5  of  the  Literature 
paper  were  rather  low.  It  seems  reasonable,  that  if  the 
student  has  only  vague  ideas  on  a  topic,  that  the  expression 
of  his  ideas  will  suffer  as  a  consequence. 

The  correlation  between  the  total  Language  mark  and 
the  total  for  Social  Studies  was  .36,  (P.E.  .040). 

Relationship  of  Principals1  Confidential  Marks 

A  further  correlation  was  worked  out  between  the  Trans¬ 
muted  Raw  Scores  or  the  Department’s  Pinal  Scores  and  the  marks 
assigned  to  each  candidate  by  the  various  school  principals 
in  their  Confidential  Reports  to  the  Department. 

These  Confidential  Marks  are  intended  to  be  the  Prin¬ 
cipal’s  estimate  of  each  candidate  writing  from  his  school  or  it 
may  be  viewed  as  a  prediction  of  ihat  the  candidate  ought  to  re¬ 
ceive  on  the  Department’s  examination.  In  this  we  find  a  high 
degree  of  agreement.  The  correlation  between  these  two  marks 
is  .7,  (P.E.  .0205).  This  might-  be  taken  as  an  indication  that 
in  general  the  examination  of  1943  attained  results  in  keeping 
with  previous  examinations.  It  Is  not  to  be  thought  that 
the  Principal’s  estimate  and  the  Department’s  findings  are 
in  agreement,  but  merely  that  the  Department  has  placed  the 
candidates  in  the  same  relative  rank  as  the  Principal  has 
done.  It  must  be  kept  in  mind  that  the  Principal  forms  his 


. 


. 


. 


, 

'  '  -.i.* 


■ 


73 


judgment  in  the  light  of  a  comparison  between  this  candidate 
and  the  attainment  of  the  candidates  presented  in  previous 
years.  This  point  may  be  a  factor  in  the  high  correlation 
which  was  found  in  this  comparison. 
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CHAPTER  VI 


SUMMARY'  OF  RESULTS  AND  CONCLUSIONS 

1.  The  Distribution  of  the  Raw  Scores  of  the  exam¬ 
ination  in  English  3  was  found  to  conform  almost  completely 
with  that  of  a  normal  distribution.  Hence,  the  opinion  that 
there  seemed  to  be  abnormal  crowding  near  the  middle  of  the 
range  is  not  substantiated. 

2.  Both  the  Part  I  A  Scores,  that  is,  the  marks 
assigned  by  the  teachers  on  the  candidates1  reading  and 
study  throughout  the  year  and  the  Score-A,  that  is,  the 
marks  obtained  by  the  candidates  on  the  Department’s  written 
tests  alone,  yield  normal  distributions.  Hence,  the  feel¬ 
ing  that  the  Part  I  £  Score  was  exerting  a  distorting 
effect  on  the  distribution  of  the  final  results  when  these 
two  parts,  Part  I  A  and  Score-A,  were  brought  together  to 
form  the  Raw  Score,  Is  not  valid. 

3.  Since  the  distribution  of  the  Part  I  A  Score  has 
a  mean  value  about  eleven  points  higher  than  that  of  the 
Score-A,  the  net  effect  of  combining  these  two  distributions 
is  to  produce  a  distribution  with  a  mean  value  about  five 
points  higher  in  the  scale  than  that  of  the  Score-A.  This 
means  that  on  the  average  the  teachers’  marks  were  such  as 
to  virtually  raise  the  candidates*  scores  on  the  written 
tests  by  approximately  five  marks. 
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4.  The  effect  of  the  Part  I  A  Score  on  the  Score-A 

in  regard  to  the  pass  mark,  is  to  raise  the  pass  mark  from  40 
to  45  and  to  raise  the  lowest  mark  of  B  category  from  46  to 
52.  This  is  in  accordance  with  the  previous  point  where  it 
was  seen  that  the  effect  produced  was  to  move  the  distribu¬ 
tion  of  Score-A  approximately  five  points  higher  in  the  scale. 
The  median  was  similarly  affected. 

5.  The  effect  of  Part  I  A  on  the  range  of  Score-A  is 
to  change  it  from  a  17-83  to  a  27-87;  that  is,  the  width  of 
the  range  was  reduced  from  67  scale  points  to  61  points. 

The  lower  end  of  the  range  was  raised  from  17  to  27  while  the 
upper  end  of  the  range  was  augmented  only  four  points  from 
83  to  87. 

The  Interquartile  Range,  the  Plus  and  Minus  Sigma 

_  I 

Range,  and  the  P90 -^(£*ange ,  remained  approximately  the  same 
size,  although,  of  course,  their  positions  were  raised  in 

the  scale. 

The  investigation  substantiates  quite  fully  that 
there  is  an  inadequate  range  of  marks,  or  an  insufficiency 
of  score  intervals  between  the  highest  and  the  lowest  marks, 
if  highly  discriminatory  grading  is  expected.  It  shows 
also  quite  clearly,  that  since  the  ranges  of  the  Raw  Score 
and  the  Score-A  are  almost  equally  narrow,  the  effect  is 
not  greatly  attributable  to  the  influence  of  the  Part  I  A 
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It  is  recommended  that,  in  order  to  avoid  some  of 
the  undesirable  effects  arising  out  of  this  inadequacy  of 
range  or  insufficiency  of  intervals,  the  examination  paper 
be  standardized  for  difficulty  on  the  basis  of  a  frequency 
curve  of  error  before  the  marking  of  the  papers  is  done. 

By  this,  it  is  meant  that  the  various  items  should  be  pro¬ 
perly  weighted  in  accordance  with  the  frequency  with  which 
it  appears  correctly  on  the  examination  answer  papers. 

This  will  produce  a  normal  distribution  of  full  range  for 
the  written  tests.  If  this  recommendation  cannot  be  im¬ 
plemented,  it  is  alternatively  suggested,  that  the  written 
tests  be  marked  on  a  possible  maximum  of  150  or  more  each, 
in  order  to  widen  the  range. 

It  is  further  recommended  that  the  distribution  of 
the  Part  I  A  marks  be  normalized  if  it  is  not  already 
normal  and  that  the  marks  be  scaled  to  give  the  same  range 

as  the  written  tests.  The  method  of  computing  the  final 

- 1 

result  should  be  changed,  so  that  the  combining  of  the 
teachers  *  marks  and  the  marks  for  the  written  examination 
'will  be  the  last  step  in  the  procedure. 

6.  A  tabulation  was  made  whereby  the  effect  of  the 
Part  I  A  mark  on  every  Score-A  might  be  read.  The  table 
presents  rather  startling  facts.  On  a  selected  group  of 
cases  (taken  near  the  point  o£  greatest  deviation  but  by 
no  means  ah  isolated  group)  It  was  shown  that  despite  the 
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fact  that  all  members  of  the  group  turned  in  written  work  which 
was  evaluated  as  being  of  the  same  calibre,  yet  their  final 
raw  scores  varied  by  as  much  as  23  marks  and  that  their 
transmuted  scores  varied  by  as  much  as  33  marks  and  that  their 
grades  varied  from  the  lowest  of  the  C'rs  to  the  highest  of 
the  A’s.  However,  the  indications  are  that  this  condition 
is  due  to  the  operation  of  chance  factors  rather  than  any 
deliberate  attempt  to  destroy  the  reliability  of  the  exam¬ 
ination,  Nevertheless,  if  the  recommendations  made  in 
paragraph  5  and  those  to  follow  in  paragraph  8  are  imple¬ 
mented,  much  of  the  undue  influence  of  the  Part  I  A  marks 
will  be  eliminated, 

7.  Evidence  that  the  marks  given  by  the  teacher  in 
Part  I  A  are  not  exerting  greatly  a  distorting  effect  on  the 
final  score  is  shown  by  the  fact  that  between  the  Score- A 
(without  the  teachers1  marks)  and  the  Raw  Score  (including 

the  teachers’  marks)  there  is  a  correlation  of  .90. 

£ 

8.  It  is  shown  that  the  marks  given  by  the  teachers 
are  tuned  to  a  higher  point  of  reference  than  those  assigned 
by  the'  Department,  In  general,  the  one  is  about  20$  higher 
than  the  other.  It  is  recommended  that  a  set  of  standard 
values  be  set  up  to  guide  the  teachers  in  forming  their 
estimates,  so  that  near  uniformity  may  be  attained  and 
secondly,  that  the  marks  so  given  be  properly  scaled  before 
being  combined  with  the  Department’s  scores. 
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9.  A  study  of  the  changes  in  the  numbers  of  can¬ 
didates  receiving  the  different  grades  when  the  grading  is 
made  on  the  basis  of  the  Raw  Score  and  when  it  is  made  on 
the  basis  of  Score-A,  reveals  that  the  effect  of  Part  I  A 
is  a  centralizing  one.  This  is  shown  by  the  reduction  in 
the  frequencies  at  the  extremes,  the  H*s,  ATs  and  D*s, 
while  the  central  ones,  the  Bfs  and  Cfs.are  increased. 
Briefly,  Part  I  A  tends  to  reduce  the  range  and  to  move  the 
mean  up  to  a  higher  point  in  the  scale. 

10.  The  special  effects  investigated  are  best  seen 
by  a  study  of  the  following  list  of  correlations. 
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TABLE  XV 


LIST  OE  CORRELATIONS 


RE. 

Literature  A  with  Literature  B 

+  .46 

.0092 

Literature  A  with  Raw  Score  of  Social  .Studies 

+  .35 

.041 

Literature  A  with  Literature  Question  5 

+  .13 

.039 

Literature  A  with  Score  -  A 

+  .51 

.029 

Language  A  with  Language  Question  5 

+  .28- 

.037 

Language  A  with  Language  Social  Studies 

+ .  23 

.044 

Language  Total  with  Social  Studies  Total 

+  .36 

.040 

Raw  Score  with  Score  -  A 

*.90 

.0076 

Part  I  A  (Lit.  A)  with  Score  -  A 

+  .51 

.029 

Dept's  Transmuted  Score  with  Confidential  Marks 

+  .7 

.0205 

. 
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HE  C  0MM1NDATI ONS 

1.  That  the  examination  marking  schedule  be  stand¬ 
ardized  and  be  used  to  give  a  normal  distribution  extending 
for  the  complete  range  of  0  -  100. 

2.  That  the  distribution  of  the  Part  I  A  marks  be 
normalized,  if  it  is  not  so  already,  and  that  the  individual 
marks  be  scaled  to  the  same  scale  as  the  written  tests. 

3.  That  the  method  of  computing  a  candidates  final 
score  be  changed  so  as  to  make  the  combining  of  the  trans¬ 
muted  score  of  the  candidates  written  tests  with  the.  Part  I 
A  marks  the  final  step  in  the  computation. 

4.  That  standards  be  set  to  guide  the  teacher  in  e- 
valuating  a  candidates  work  for  the  purpose  of  the  Part  I 
A  marks. 

5.  That,  in  the  event  that  recommendation  number  one 
above  is  not  implemented,  the  written  tests  be  marked  on'  a 
much  wider  scale  such  as  150. 
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APPENDIX  A 

THE  FORMAT  OF  THE  ENGLISH  3  EXAMINATION 

The  format  of  the  examination  in  English  3  will  be  made 
clear  by  a  reference  to  the  Regulations  of  the  Department  of 
Education"  from  which  the  following  is  quoted. 1 

"(i)  The  examination  in  English  3  consists  of  two  parts: 
Part  I  on  Reading  and  Literature,  and  Part  II  on  Language. 

"  The  student’s  score  for  Part  I  comprises  two  parts, 

"A"  and  r’B".  The  Part  I  "A"  score  is  given  on  the  basis  of  a 
special  report  from  the  instructor  in  Reading  and  Literature, 
certifying  the  student’s  work  in  the  reading  and  study  of  Poetry, 
Drama,  and  Essays  and  in  Free  Reading,  in  accordance  with  the 
authorized  list  as  set  forth  in  Bulletin  II  of  the  Programme  of 
Studies  for  the  High  School.  The  special  report  must  show 
whether  or  not  the  student’s  work  has  been  consistently  and 
continuously  satisfactory  throughout  the  year. 

"  The  Part  I  "B"  score  is  given  on  the  results  of  a 
written  test,  mainly  of  reading  comprehension  and  literary 
appreciation,  to  be  set  on  "sight"  selections.  There  will 
be  no  attempt  to  examine  specifically  on  selections  from  the 
prescribed  or  authorized  reading;  but  a  few  general  questions 
on  this  reading  may  be  given  along  with  the  questions  on 

^-Programme  of  Studies  for  the  High  School,  Regulations  of  the 
Department  of  Education,  1944,  page  28. 
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"sight"  selections. 

"Part  II  of  the  examination  in  English  3  is  a  test  on 
Language.  Of  the  final  possible  score,  30  per  cent  will  be 
derived  from  the  evaluation  for  English  Language  of  certain 
questions  of  Part  I  "B";  and  70  per  cent  will  come  from 
questions  on  language  usage,  and  from  questions  of  types 
denoted  by  the  term  "composition,"  including  an  essay. 

This  essay  will  be  evaluated  for  English  Language  with  much 
the  same  procedure  as  that  applied  to  the  evaluation  of 
certain  questions  of  Part  I  "B",  as  mentioned  above. 

"  (ii)  The  general  procedures  of  evaluation  for 
English  Language  will  be  the  same  as  those  which  have  been 
used  during  the  last  two  years  for  the  Grade  IX  Examinations. 
It  is  very  important  to  note  that  one  or  more  of  the  can¬ 
didates1  answer  papers  in  subjects  other  than  English  will 
be  evaluated  also  for  English  language  by  means  of  the 
procedures  just  mentioned.2  All  evaluations  for  English 
Language  will  be  made  by  a  special  group  of  sub- examiners, 
and  not  by  those  who  score  for  content. 

"  (iii)  The  following  schedule  shoves  the  method 
of  determining  a  candidates  final  score  in  English  3: 


2In  1943,  The  Social  Studies  Examination  was  tested  for 
English. 
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Part  I  ”A”:  Instructor’s  report  -  possible  percentage  ..  100 


Part  I  MB” :  Score  on  written  test  -  final  possible  .  100 

200 

Final  possible  score  on  Part  I  -  jjr  of  200  .  100 

Part  II:  Final  possible  score  . .  100 

200 

■S 


Final  possible  score  in  English  3:  i  of  200  *  100 

M  From  this  schedule  it  can  be  seen  that  the  possible 
score  for  Part  I  ’’A”  is  one-fourth  of  the  final  possible  score 
in  English  3”. 
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